1. Field of the Invention
The present invention relates to image processing and a device.
2. Brief Description of Related Developments
In recent years, digital technology has developed intensely, so various methods have been developed to transmit and record various images in digital form. Advantages of digital technology include better reproduced, disturbance-free and enduring images; also deterioration of quality can be avoided in copying. In a digital recording device, also data of the contents of the recorded image can be included, which facilitates the retrieval of the desired image. However, the implementation of digital technology in image recording and transmission has been decelerated e.g. by the large quantity of data required by the digital presentation form.
The quantity of image information increases in relation to the resolution of image accuracy, i.e. as the accuracy is reduplicated the quantity of information is typically quadruplicated. The image information is composed of pixels, each of which is showing one detail of the image. This image information is formed e.g. of the luminance data of the pixel and in color images also of the color data of the pixel, as explained later in this specification.
The resolution of the image can be e.g. 256×256, i.e., the image is advantageously divided into 256 horizontal portions and 256 vertical portions. In case of a black and white image with 256 grey tone values the image information of each pixel can be illustrated as eight binary digits, i.e., by one byte. Thus, the quantity of image information without compression is 524 288 bits (=256×256×8 bits), i.e., 65 536 bytes.
An image according to VGA standard (Video Graphics Array) which is known as such e.g. in connection with computers, is composed of 640 horizontal pixels and 480 vertical pixels, e.g. the image resolution is 640×480. In an image in accordance with a more recent, XGA standard (Extended Graphics Array), the resolution can even exceed 1,000×1,000 pixels, wherein one digitized, uncompressed image comprises over a million bytes.
In many embodiments it is, however, not necessary to show such large pictures. E.g. a telephone book embodiment, where e.g. a graphic display of a mobile station can, in addition to the telephone number, display the picture of the person concerned, can be implemented by a relatively rough resolution and still the person is recognizable on the basis of the picture.
Numerous different methods have been developed for compressing image information in electric form (digitized). One known compression method is disclosed in the ISO standard DIS 10918-1 of the International Organisation for Standardization, i.e., in so-called JPEG standard (Joint Photographic Experts Group). The JPEG standard is designed for compression of color and grey tone images taken of naturalistic image objects. Compression according to the JPEG standard is most efficient in compression of photographs, naturalistic paintings, computer formed pictures which aim to look naturalistic, as well as other corresponding images.
Compression according to the JPEG standard is not necessarily loss-free, i.e., the compressed image can for some parts be different than the original image. These changes are usually such small changes which cannot be detected by the human eye. By using compression in accordance with JPEG standard, it is possible to obtain compression ratio of the value 10:1 to 20:1 without observable losses, i.e., the size of the compressed image is less than 10% of the original image size. If some minor, visually observable changes are tolerated in the image, even a compression ratio of 50:1 can be obtained.
The JPEG standard is illustrated in more detail e.g. in Reference 1: “JPEG Still Image Data Compression Standard” and in Reference 2: “Digital Compression and Coding of Continuous-tone Still Images—Part 1”. A more detailed list of reference literature is found in the end of the present specification.
In order to apply the compression method in accordance with the JPEG standard, the image, or any other data signal treated in accordance with the method, has to be in digital form. A generally used color image signal is composed of luminance and chrominance data. When the image is converted into digital form, both the luminance and the chrominance data is converted separately, usually in an accuracy of 8 bits.
Color images can be formed e.g. in a manner that the information of each pixel is composed of luminance and chrominance information. Thus, the quantity of information increases to triple, in case the chrominance information is illustrated by a bit quantity corresponding to that of the luminance information. Due to the properties of the human eye, the chrominance information can, however, be illustrated by using a smaller accuracy than the luminance information, in a manner that the eye does not detect any deterioration in image quality. Typically, both components of the chrominance information are formed from a square comprising four pixels. Color images can also be illustrated by a so-called RGB image in which each pixel is illustrated by using three primary colors, i.e., red (R), green (G) and blue (B). Thus, the quantity of information is in one image triple compared to a grey tone image. In a color image the intensity of each color is usually divided into 256 portions in a RGB image, i.e., 24 bits (3 bytes) are required for showing one pixel.
Usually also RGB images are converted into luminance/chrominance form before the image is compressed. For this purpose, a widely used conversion formula has been developed, which is known as abbreviation CCIR60 and marked as YCbCr. This is a kind of three dimensional presentation form, in which the various components can be calculated from the R, G and B information in the following manner:Y=0.299R+0.587G+0.114B  (1a)Cb=−0.168R−0.3313G+0.5B  (1b)Cr=0.5R−0.4187G−0.0813B  (1c)
The luminance component Y illustrates the grey tones of the image and it can be used when showing a black and white image and when showing a color image in black and white. There are two chrominance components: Cb and Cr, which include the color information of the image. However, a color image can also be compressed in RGB form, but in this case the same compression efficiency that is obtained when compressing an image in luminance/chrominance form cannot usually be obtained.
In various image compression techniques, a discrete cosine transform (DCT) is performed to an image signal that has been converted to digital form, before the image signal is transmitted to a data transfer medium or recorded into recording media. By the DCT conversion it is possible to calculate the frequency spectrum of the periodic signal, i.e., to shift from time domain to frequency domain. The word discrete denotes in this context to the fact that in the conversion separate pixels are treated instead of continuous variables. In the image signal, the successive pixels have typically a large mutual correlation. One property of the DCT conversion is that the facients generated as a result of the DCT conversion are practically taken uncorrelated, so that the DCT conversion performs efficiently the conversion of the image signal from time domain to frequency domain.
When a discrete cosine transform is used in compression of a single image, two-dimensional conversion is required. Instead of time, the variables are the latitude and altitude co-ordinates X and Y. Further, the frequency is not the quantity of sequences in a second, as normally, but it illustrates e.g. the conversion rate of the luminance in the direction of location co-ordinates X,Y. This is called spatial frequency.
An image comprising a large amount of micronic details shows large spatial frequencies. E.g. the parallel lines in the image correspond to the larger frequency the more densely they are located. Diagonal directed frequencies that are larger than a certain limit value can be quantized more in image processing without observably deteriorating the image quality.
In the JPEG compression, the DCT conversion is carried out in blocks in a manner that the block size is 8×8 pixels. The luminance level that is converted has the full resolution. Both chrominance signals are sub-sampled, e.g. an area of 16×16 pixels is subsampled to an area of 8×8 pixels. The differences in block sizes are by and large due to the fact that the eye does not detect changes in chrominance as easily as it detects changes in luminance, wherein an area of 2×2 pixels is coded with the same chrominance value. However, the invention is not restricted merely to the JPEG compression and said block sizes, but it can be adapted also in other corresponding compression methods and in blocks of various sizes.
For example when converting luminance values to the DCT level, the luminance values and horizontal and vertical spatial frequencies calculated thereof are brought from-the pixel block that is converted. Each frequency component is calculated of all the values of the block that is converted. Thus, the elements of the coefficient matrix that are obtained by discrete cosine transform do not correspond to single pixels of the block that is converted. In the coefficient matrix, the horizontal frequency components illustrate the horizontal changes in the block that is converted, and in a corresponding manner, the vertical frequency components illustrate the vertical changes in the block that is converted. The first element in the first row in the upper left corner of the matrix illustrates the zero frequency value of the image block that is converted, because it is comparable to the average of the pixels of the block that is converted.
Subsequent to the calculation of the coefficient matrix, i.e., after the DCT conversion, a quantization is performed to an element F(i,j) of a coefficient matrix F, i.e., the elements are divided into quantization levels of an appropriate size in a manner that the visual system of the human eye is taken into account. FIG. 1a shows an example of a luminance-signal quantization matrix QL used generally in image quantization, and FIG. 1b shows an example of a chrominance-signal quantization matrix QC used generally in image quantization. Quantization is performed advantageously in accordance with Formula (2a). In image dequantization, e.g. when receiving a compressed image from data transfer medium, in the inverse transformation (iDCT) that is performed, a similar quantization matrix QL, QC are used in accordance with Formula (2b). Based on the quantization matrices QL, QC and formulas (2a) and (2b), it can be detected that in connection with larger frequencies and in diagonal direction fewer quantization levels are used than in connection with frequencies close to zero. This is especially due to the fact that larger diagonal frequencies are less important to the human visual system than frequencies close to zero frequency and substantially horizontal and vertical frequencies.                               QF          ⁡                      (                          i              ,              j                        )                          =                              F            ⁡                          (                              i                ,                j                            )                                            Q            ⁡                          (                              i                ,                j                            )                                                          (2a)            Rec(i,j)=Q(i,j)×QF(i,j)  (2b)
Subsequently, for a quantized, DCT converted matrix, a coding is performed, wherein at first the elements of each matrix are arranged sequentially, preferably in a manner that the first element is an matrix element (0,0). Next, the second element (0,1) in the first horizontal line of the matrix is selected. The next step is to move in diagonal direction down and to the left to the first element (1,0) of the second horizontal line. From here it is moved one line down to the first element (2,0) of the third horizontal line, and thereafter back in the diagonal direction, up and right to the first horizontal line, i.e., in the order (0,0), (0,2). Finally, the last element in the last horizontal line is placed, which in connection with the most usually used 8×8 block size is the eighth element of the eighth horizontal line (7,7). The aim of this arrangement is to take into account e.g. the fact that the images usually contain more information in smaller frequencies than in larger frequencies, wherein particularly the coefficients illustrating larger frequencies are zero in connection with many images. Thus, several sequential zero values are obtained which can be replaced by information showing the quantity of successive zero values. FIG. 1g further illustrates by means of a line drawn in the matrix the sequential arrangement of the matrix elements.
In order to code the quantized, DCT converted image, the JPEG standard illustrates two coding systems: Huffman coding and arithmetic coding. These methods transform the data that is converted into codes of varying length in a manner, that for often repeating symbols is formed a shorter code word than for less frequently repeating symbols. In addition, in Huffman coding, no code word appears in the initial part of another code word. When e.g. the code word that corresponds to a bit string (symbol) ‘0010 0001’ is ‘10’, then no other code word begins with the bits ‘10’.
FIG. 1c illustrates a Huffman Table H, which has proved to be efficient in image compression. Table of FIG. 1c is regarded for coding the DC difference value of a luminance signal. The compression efficiency of this table has been compared to optimal compression, performed by a table calculated from the image information by Huffman coding (Reference 3, a list of references is in the end of the specification). In the comparison, different images were used, a Huffman table being calculated for each image. The size of the compressed image was typically only less than 5 percent larger when using the table of FIG. 1c than when using the optimal table calculated from the image information. On the other hand, to calculate a compression table for each image separately increases the capacity required for the compression device, wherein in a large part of JPEG coded images a Huffman coding table of FIG. 1c have been used. The operation of Huffman coding is illustrated later in this specification when an apparatus according to the advantageous embodiment of invention is discussed.
Advantageously after the coding, a data frame is formed wherein e.g. the coded image information is located. FIG. 3a illustrates in reduced form a data frame 300 in accordance with the JPEG standard. The data frame 300 comprises header data 301 and data field 302, wherein the actual image information is located. The header data 301 contain e.g. the following fields: a start of image (SOI) 303 of the frame, a JPEG file interchange format header (JFIF) 304, one or more Huffman coding tables 305, and a quantization table 306, a start of frame (SOF0) 307, and a start of scan (SOS) 308 of the image. Next in the data frame is the image information, which is compressed advantageously by using DCT conversion and coded by Huffman coding. Subsequent to the image information there is still an end of image (EOI) field 309, which informs of the end of the data frame. It is known as such that also enciphering can be carried out to the data frame, if necessary, by using an encryption algorithm or an encryption key.
FIG. 3a illustrates also the length of each field as bytes. The length of the header data is in this example 424 bytes. The length of the image information depends e.g. on the resolution of the image and the coding and compression level used in the compression of the image. In connection with portable electric devices, such as mobile stations, the images are usually rather small, wherein the length of the header data forms a relatively large proportion of the entire data frame.
The need to process images will increase also in different portable devices, such as mobile stations. The drawback is then that images compressed in the present form in accordance with prior art require a large quantity of storage capacity. This restricts the quantity of images that are storable into a portable device at a time and, on the other hand, also the image transmission lasts relatively long.
At the time when the compressed image is wished to be converted into uncompressed form, e.g. for viewing, the image is Huffman decoded, dequantized and an inverse DCT conversion (iDCT) is carried out. To perform the decoding, it is necessary to know the contents of the coding table and, in a corresponding manner, to perform decompression, one has to know the contents of the table used in compression. In image compression systems of prior art these tables which are read in the header field in the inverse conversion of the image are added in the header of the data frame. However, this involves the drawback that the size of the tables can be tens, even hundreds of bytes, wherein the proportion of the tables of the header field can be even larger than the quantity of the actual image information.